Why textual data?

• textual data are everywhere
• textual data are (very) promising

Two general approaches

1. Measurement
2. Prediction

One big problem

• text data \(\neq\) numerical data
• compare this to survey data, trading data, crime statistics, etc.
• a piece of text data is just that, “a text”
• but: for statistical analyses (and statistical learning), we rely on numerical representations

The first challenge of NLP is text quantification.

This course

An introduction to statistical natural language processing in R

• the basics of handling textual data in R
• fundamental approaches of (real) NLP
• text representations
• measuring constructs in textual data
• textual data for prediction (supervised machine learning)

Structure

• short theoretical parts (4x)
• followed by practicals (4x)
• discussion of solutions

All materials: https://github.com/ben-aaron188/snlp

Ask all your questions

The quanteda package

library(quanteda)
packageVersion("quanteda")

## [1] '4.1.0'

• quanteda: Quantitative Analysis of Textual Data[REFREF]
  • documentation
  • tutorials
  • examples

How would you describe this text?

The whole situation makes me very anxious. I do not believe, as a country, we are doing everything to prevent the spread. I believe we should be in a full and proper lockdown like other countries. I fear for my children and their futures.

Different aspects of text data

• lexical dimension (text metrics)
• syntactic dimension
• semantic dimension

Lexical dimension

Numerical summaries of units of a document or collection of documents.

Examples:

• no. of words
• average sentence length

Syntactic dimension

Information about the arrangement/structure of language.

Examples:

• grammatical function of words: verbs, nouns (parts-of-speech)
• named entities: persons, locations, organisations
• structure of a sentence (parse tree)

Semantic dimension

Information of the text concerned with meaning.

• sentiment
• psycholinguistic features
• relationship between words
• contextual variation of words
• higher-order constructs

What can we work with?

1. A corpus is a collection of documents
2. A document is a collection of sentences
3. A sentence consists of words
4. A word consists of syllables
5. A syllable consists of characters

Special units in NLP

• tokens: a sub-unit of text that is the at the core of NLP
• ngrams: a sequence of \(n\) tokens

Example sentence:

The whole situation makes me very anxious.

Tokenisation

text_1 = 'The whole situation makes me very anxious.'

toks = tokens(text_1)
toks

## Tokens consisting of 1 document.
## text1 :
## [1] "The"       "whole"     "situation" "makes"     "me"        "very"     
## [7] "anxious"   "."

Common preprocessing steps

Removing punctuation:

toks_no_punct = tokens(text_1
              , remove_punct = T
              , remove_numbers = F
              , remove_url = F
              , remove_separators = F)
toks_no_punct

## Tokens consisting of 1 document.
## text1 :
##  [1] "The"       " "         "whole"     " "         "situation" " "        
##  [7] "makes"     " "         "me"        " "         "very"      " "        
## [ ... and 1 more ]

Note on tokenisation

This is far from a solved problem:

• many tokeniser algorithms exist
• each has a slightly different token segmentation approach

text_2 = 'evidence-based self-fulfilling prophesy'

tokens(text_2, what = 'word')

## Tokens consisting of 1 document.
## text1 :
## [1] "evidence-based"  "self-fulfilling" "prophesy"

Slightly different tokenisation

tokens(text_2, split_hyphens = T)

## Tokens consisting of 1 document.
## text1 :
## [1] "evidence"   "-"          "based"      "self"       "-"         
## [6] "fulfilling" "prophesy"

Other text units

# sentences
length(tokens(toks, what = 'sentence'))

## [1] 1

# tokens
ntoken(toks)

## text1 
##     8

# characters
nchar(text_1)

## [1] 42

Key idea

• summarise key information of text data numerically
• largely pertains to meta features of documents

Common text metrics: readability (see quanteda.textstats), lexical diversity

Take two examples

Example 1:

I am very anxious about this situation. Very anxious.

Example 2:

I am very anxious about this situation. Really scary.

Which has a higher lexical diversity?

Types and tokens

ex_1 = 'I am very anxious about this situation. Very anxious.'
ex_2 = 'I am very anxious about this situation. Really scary.'

• Tokens: a sub-unit of text
• Types: unique tokens

The type-token ration (TTR) is often used as a measure of lexical diversity.

TTR for our examples

ttr_1 = ntype(tokens(ex_1))/ntoken(tokens(ex_1))
ttr_1

##     text1 
## 0.8181818

ttr_2 = ntype(tokens(ex_2))/ntoken(tokens(ex_2))
ttr_2

##     text1 
## 0.9090909

The stringr package

• most powerful tool for most text string queries
• based on regex (regular expressions)
• useful cheat sheet

The corpus object

A collection of documents is a corpus.

Quanteda assumes you work with corpus objects.

• can be constructed from individual strings (= character vectors)
• or from a data.frame (or data.table)

A corpus from character vectors

corpus_1 = corpus(c(text_1, text_2))

A corpus from a data.table

data_1 = fread('/Users/bennettkleinberg/GitHub/snlp/data/real_world_worry_waves_dataset.csv')

names(data_1)[1:13]

##  [1] "id"                   "prolific_id"          "worry_wave1"         
##  [4] "chosen_emotion_wave1" "anger_wave1"          "disgust_wave1"       
##  [7] "fear_wave1"           "anxiety_wave1"        "sadness_wave1"       
## [10] "happiness_wave1"      "relaxation_wave1"     "desire_wave1"        
## [13] "text_long_wave1"

corpus_rwwd = corpus(data_1$text_long_wave1)

corpus_rwwd

## Corpus consisting of 1,152 documents.
## text1 :
## "I am more worried about getting access to my normal healthca..."
## 
## text2 :
## "I am a little concerned that so many people are not staying ..."
## 
## text3 :
## "I feel relaxed. I'm aware of the risks and I'm doing my very..."
## 
## text4 :
## "I feel a little stressed by it all, I would like to go outsi..."
## 
## text5 :
## "I am fed up about not being able to go out when I want, I mi..."
## 
## text6 :
## "At this very moment, I am feeling very fearful and sad about..."
## 
## [ reached max_ndoc ... 1,146 more documents ]

summary(corpus_rwwd, n = 10)

## Corpus consisting of 1152 documents, showing 10 documents:
## 
##    Text Types Tokens Sentences
##   text1    89    132         6
##   text2    70    106         5
##   text3    72    114         6
##   text4    71    108         4
##   text5    71    117         5
##   text6    68    120         6
##   text7    73    118         2
##   text8    65    109         5
##   text9    91    136         3
##  text10    88    124         4

Now: PRACTICAL I

• becoming comfortable with text handling in R
• calculating text metrics on real data

Aim of text representations

A numerical representation of textual data.

Most common approach:

• representing a text by its tokens
• each text consists of a frequency of its tokens

By hand

I think I believe him

How can we construct a term-frequency table?

Steps

1. create a column for each token
2. count the frequency

Term frequency

• frequency of each token in each document
• represented in a table (matrix)
• tokens are features of a document
• called a Document Feature Matrix (= DFM)

The DFM in quanteda

• from tokens or corpus objects, create a DFM table

ex_3 = 'I think I believe him'

corpus_ex_3 = corpus(ex_3)
tokens_ex_3 = tokens(corpus_ex_3)

dfm(tokens_ex_3
    , tolower = T)

## Document-feature matrix of: 1 document, 4 features (0.00% sparse) and 0 docvars.
##        features
## docs    i think believe him
##   text1 2     1       1   1

Sparsity

Sparsity = % of zero-cells

• Why is the sparsity 0% here?
• What would you expect if we take additional documents, and why?

DFM with multiple documents

corpus_3 = corpus(c("I think I believe him", "This is a cool function"))
tokens_corpus_3 = tokens(corpus_3)

dfm(tokens_corpus_3)

## Document-feature matrix of: 2 documents, 9 features (50.00% sparse) and 0 docvars.
##        features
## docs    i think believe him this is a cool function
##   text1 2     1       1   1    0  0 0    0        0
##   text2 0     0       0   0    1  1 1    1        1

Remember n-grams?

What happens if we add bi-grams (i.e., \(n=1\) and \(n=2\))?

I think I believe him

ngrams in R

tokens_ngrams(tokens(ex_3), n = 1:2)

## Tokens consisting of 1 document.
## text1 :
## [1] "I"           "think"       "I"           "believe"     "him"        
## [6] "I_think"     "think_I"     "I_believe"   "believe_him"

So we very quickly run into dimensionality problems with text data.

The Real-World Worry Dataset corpus

Unigrams (~ tokens)

tokens_corpus_rwwd = tokens(corpus_rwwd)

dfm_unigrams = dfm(tokens_ngrams(tokens_corpus_rwwd
                                 , n = 1)
                   , tolower = T)

dfm_unigrams

## Document-feature matrix of: 1,152 documents, 6,274 features (98.71% sparse) and 0 docvars.
##        features
## docs    i am more worried about getting access to my normal
##   text1 7  3    1       1     2       1      1  5  5      1
##   text2 3  2    0       1     0       0      0  3  0      0
##   text3 6  0    0       0     2       0      0  5  3      0
##   text4 4  0    0       0     0       0      0  6  1      2
##   text5 7  2    1       1     1       0      0  7  4      0
##   text6 7  3    0       1     4       0      0  2  3      0
## [ reached max_ndoc ... 1,146 more documents, reached max_nfeat ... 6,264 more features ]

Unigrams, bigrams and trigrams

dfm_ngrams = dfm(tokens_ngrams(tokens_corpus_rwwd
                                 , n = 1:3)
                   , tolower = T)

dfm_ngrams

## Document-feature matrix of: 1,152 documents, 150,549 features (99.79% sparse) and 0 docvars.
##        features
## docs    i am more worried about getting access to my normal
##   text1 7  3    1       1     2       1      1  5  5      1
##   text2 3  2    0       1     0       0      0  3  0      0
##   text3 6  0    0       0     2       0      0  5  3      0
##   text4 4  0    0       0     0       0      0  6  1      2
##   text5 7  2    1       1     1       0      0  7  4      0
##   text6 7  3    0       1     4       0      0  2  3      0
## [ reached max_ndoc ... 1,146 more documents, reached max_nfeat ... 150,539 more features ]

dfm_ngrams[1:10, 201:210]

## Document-feature matrix of: 10 documents, 10 features (86.00% sparse) and 0 docvars.
##        features
## docs    a_walk walk_, ,_but but_that that_is is_not not_possible possible_right
##   text1      1      1     1        1       1      1            1              1
##   text2      0      0     0        0       0      0            0              0
##   text3      0      0     0        0       0      0            0              0
##   text4      0      0     0        0       0      1            0              0
##   text5      0      0     0        0       0      0            0              0
##   text6      0      0     0        0       0      0            0              0
##        features
## docs    right_now now_.
##   text1         1     1
##   text2         0     0
##   text3         0     0
##   text4         0     0
##   text5         0     0
##   text6         0     0
## [ reached max_ndoc ... 4 more documents ]

High-dimensionality is undesirable

• adds little information
• statistically problematic
• computationally inefficient

But: not all words (ngrams) are equally important.

Most common terms

topfeatures(dfm_ngrams, n = 20)

##     .     i   the   and    to     ,   ._i    my    of     a    is    am  that 
##  6464  6049  5842  5234  4369  2966  2344  2289  2091  2035  1928  1775  1769 
##    it  i_am    in   for    as about  this 
##  1715  1669  1628  1567  1429  1410  1346

These add little information.

Also known as stopwords and often removed in preprocessing.

Stopwords

Common stopword lists:

stopwords(language = 'en')[1:20]

##  [1] "i"          "me"         "my"         "myself"     "we"        
##  [6] "our"        "ours"       "ourselves"  "you"        "your"      
## [11] "yours"      "yourself"   "yourselves" "he"         "him"       
## [16] "his"        "himself"    "she"        "her"        "hers"

Note: no universal agreement on stopwords!

Removing stopwords

tokens_rwwd = tokens_select(tokens_corpus_rwwd
                            , pattern = stopwords("en")
                            , selection = "remove")

dfm(tokens_ngrams(tokens_rwwd
                  , n = 1:3)
    , tolower = T)

## Document-feature matrix of: 1,152 documents, 113,954 features (99.85% sparse) and 0 docvars.
##        features
## docs    worried getting access normal healthcare moment , concerned corona .
##   text1       1       1      1      1          1      1 6         1      1 5
##   text2       1       0      0      0          0      0 2         1      0 5
##   text3       0       0      0      0          0      0 3         0      0 6
##   text4       0       0      0      2          0      0 5         0      0 4
##   text5       1       0      0      0          0      0 4         0      0 4
##   text6       1       0      0      0          0      1 2         0      1 6
## [ reached max_ndoc ... 1,146 more documents, reached max_nfeat ... 113,944 more features ]

Special language-based dimensionality reduction

Take these example:

1. Tokens: creating, creates, create
2. Tokens: gone, went, going, goes

These all belong to the same “core”.

Stemming and lemmatisation can solve this.

Stemming

Idea: reduce a word to its word stem. Works by removing suffixes.

toks

## Tokens consisting of 1 document.
## text1 :
## [1] "The"       "whole"     "situation" "makes"     "me"        "very"     
## [7] "anxious"   "."

tokens_wordstem(toks)

## Tokens consisting of 1 document.
## text1 :
## [1] "The"     "whole"   "situat"  "make"    "me"      "veri"    "anxious"
## [8] "."

tokens_wordstem(tokens(c('creating creates create')))

## Tokens consisting of 1 document.
## text1 :
## [1] "creat" "creat" "creat"

tokens_wordstem(tokens(c('gone went going goes')))

## Tokens consisting of 1 document.
## text1 :
## [1] "gone" "went" "go"   "goe"

Lemmatisation

Idea: reduce a word to its root meaning (the lemma)

library(textstem)
lemmatize_words(c('gone', 'went', 'going', 'goes'))

## [1] "go" "go" "go" "go"

#lemmatize_words(c('better', 'best'))

Impact of stemming

The RWWD dataset:

• original ngram representation: 150,549 features (99.79% sparse)
• after stopword removal: 113,954 features (99.85% sparse)

dfm_stopw = dfm(tokens_ngrams(tokens_rwwd
                  , n = 1:3)
                , tolower = T)
dfm_red = dfm_wordstem(x = dfm_stopw)
dfm_red

## Document-feature matrix of: 1,152 documents, 106,412 features (99.84% sparse) and 0 docvars.
##        features
## docs    worri get access normal healthcar moment , concern corona .
##   text1     1   2      1      1         1      1 6       1      1 5
##   text2     1   1      0      0         0      0 2       1      0 5
##   text3     0   0      0      0         0      0 3       1      0 6
##   text4     0   2      0      2         0      0 5       0      0 4
##   text5     1   1      0      0         0      0 4       0      0 4
##   text6     1   0      0      0         0      1 2       0      1 6
## [ reached max_ndoc ... 1,146 more documents, reached max_nfeat ... 106,402 more features ]

Further reducing sparsity

We still have high sparsity: 106,412 features (99.84% sparse)

Ideas?

Sparsity correction

Trimming the DFM based on term- or document-frequencies:

dfm_red_2 = dfm_trim(dfm_red
                , min_docfreq = 0.05
                , docfreq_type = 'prop')
dfm_red_2

## Document-feature matrix of: 1,152 documents, 260 features (87.51% sparse) and 0 docvars.
##        features
## docs    worri get normal moment , concern corona . health even
##   text1     1   2      1      1 6       1      1 5      1    1
##   text2     1   1      0      0 2       1      0 5      0    0
##   text3     0   0      0      0 3       1      0 6      0    0
##   text4     0   2      2      0 5       0      0 4      0    0
##   text5     1   1      0      0 4       0      0 4      0    0
##   text6     1   0      0      1 2       0      1 6      2    0
## [ reached max_ndoc ... 1,146 more documents, reached max_nfeat ... 250 more features ]

A remaining problem…

We assume all words are equally important.

But:

• some words occur all by the nature of language (e.g., stopwords)
• or by the nature of the specific context (here: worries about COVID-19)

We want to:

• weigh words so that we
  • ‘reward’ local importance (within documents)
  • ‘punish’ for global occurrences (across documents)

Weighting of words

Metric for word importance: frequency of term \(t\) in document \(d\)

• \(tf(t, d) = \sum_{t \in d} t\)
• \(tf(t, d) = \frac{\sum_{t \in d} t}{nwords_d}\) (proportion)

Term proportions

dfm_weight(dfm_red_2
           , scheme = 'prop')[1:10, 1:10]

## Document-feature matrix of: 10 documents, 10 features (54.00% sparse) and 0 docvars.
##        features
## docs         worri        get     normal     moment          ,    concern
##   text1 0.02040816 0.04081633 0.02040816 0.02040816 0.12244898 0.02040816
##   text2 0.02222222 0.02222222 0          0          0.04444444 0.02222222
##   text3 0          0          0          0          0.07894737 0.02631579
##   text4 0          0.04081633 0.04081633 0          0.10204082 0         
##   text5 0.02325581 0.02325581 0          0          0.09302326 0         
##   text6 0.01694915 0          0          0.01694915 0.03389831 0         
##        features
## docs        corona          .     health       even
##   text1 0.02040816 0.10204082 0.02040816 0.02040816
##   text2 0          0.11111111 0          0         
##   text3 0          0.15789474 0          0         
##   text4 0          0.08163265 0          0         
##   text5 0          0.09302326 0          0         
##   text6 0.01694915 0.10169492 0.03389831 0         
## [ reached max_ndoc ... 4 more documents ]

Term counts

dfm_weight(dfm_red_2
           , scheme = 'count')[1:10, 1:10]

## Document-feature matrix of: 10 documents, 10 features (54.00% sparse) and 0 docvars.
##        features
## docs    worri get normal moment , concern corona . health even
##   text1     1   2      1      1 6       1      1 5      1    1
##   text2     1   1      0      0 2       1      0 5      0    0
##   text3     0   0      0      0 3       1      0 6      0    0
##   text4     0   2      2      0 5       0      0 4      0    0
##   text5     1   1      0      0 4       0      0 4      0    0
##   text6     1   0      0      1 2       0      1 6      2    0
## [ reached max_ndoc ... 4 more documents ]

Correcting for global occurrence

• Number of documents \(d\) with term \(t\)
• \(df(d, t) = \sum_{d \ni t} 1\)

docfreq(dfm_red_2
        , scheme = 'count')[1:10]

##   worri     get  normal  moment       , concern  corona       .  health    even 
##     673     487     216     184     894     213     104    1138     272     146

Simple document frequency

Correcting for term inflation

• we want: low values for common words
• and: higher values for important words

Trick: inverse document-frequency

Inverse DF (IDF)

\(idf(t) = \frac{n}{df(d, t)}\) for \(n\) documents in the corpus.

IDF

Problem?

When \(n\) becomes really large…

Solution: log IDF

• to avoid extreme values, we use the logarithm
• simple transformation: \(idf(t) = log(\frac{n}{df(d, t)+1})\)

Log IDF

Combining term frequency and inverse document frequency (TF-IDF)

1. Local importance (TF)

dfm_weight(dfm_red_2, scheme = 'count')[1:10, 1:10]

## Document-feature matrix of: 10 documents, 10 features (54.00% sparse) and 0 docvars.
##        features
## docs    worri get normal moment , concern corona . health even
##   text1     1   2      1      1 6       1      1 5      1    1
##   text2     1   1      0      0 2       1      0 5      0    0
##   text3     0   0      0      0 3       1      0 6      0    0
##   text4     0   2      2      0 5       0      0 4      0    0
##   text5     1   1      0      0 4       0      0 4      0    0
##   text6     1   0      0      1 2       0      1 6      2    0
## [ reached max_ndoc ... 4 more documents ]

2. Correct for global occurrences (IDF)

docfreq(dfm_red_2
        , scheme = 'inverse'
        , k = 1)[1:10]

##       worri         get      normal      moment           ,     concern 
## 0.232792583 0.373032657 0.724992745 0.794280751 0.109629444 0.731038706 
##      corona           .      health        even 
## 1.040263180 0.004928755 0.625289832 0.894135144

Full flow in R for TF-IDF weighting

dfm_tfidf = dfm_tfidf(dfm_red_2
                      , scheme_tf = "count"
                      , scheme_df = "inverse")

dfm_tfidf

## Document-feature matrix of: 1,152 documents, 260 features (87.51% sparse) and 0 docvars.
##        features
## docs        worri       get    normal    moment         ,   concern   corona
##   text1 0.2334374 0.7478470 0.7269987 0.7966347 0.6606898 0.7330729 1.044419
##   text2 0.2334374 0.3739235 0         0         0.2202299 0.7330729 0       
##   text3 0         0         0         0         0.3303449 0.7330729 0       
##   text4 0         0.7478470 1.4539975 0         0.5505748 0         0       
##   text5 0.2334374 0.3739235 0         0         0.4404598 0         0       
##   text6 0.2334374 0         0         0.7966347 0.2202299 0         1.044419
##        features
## docs             .    health      even
##   text1 0.02655109 0.6268836 0.8970996
##   text2 0.02655109 0         0        
##   text3 0.03186130 0         0        
##   text4 0.02124087 0         0        
##   text5 0.02124087 0         0        
##   text6 0.03186130 1.2537672 0        
## [ reached max_ndoc ... 1,146 more documents, reached max_nfeat ... 250 more features ]

Text similarity

• each text can be represented as a vector
• e.g., simple DFM or a TF-IDF weighted DFM